TaskGroups — Organizing Large DAGs Visually

The Problem

Picture a DAG that extracts from three sources and loads into two targets. Without any grouping, that's five tasks scattered flat across the Graph View, with no visual signal that "these three belong together" or "these two belong together." At 5 tasks it's mildly annoying. At 40, it's unreadable.

Grouping Tasks with TaskGroup

from airflow.decorators import dag, task
from airflow.utils.task_group import TaskGroup
from datetime import datetime


@dag(schedule=None, start_date=datetime(2024, 1, 1), catchup=False)
def multi_source_pipeline():

    @task()
    def start():
        print("Pipeline started")

    with TaskGroup(group_id="extract_sources") as extract_group:
        @task()
        def extract_api():
            return {"source": "api", "rows": 500}

        @task()
        def extract_db():
            return {"source": "db", "rows": 1200}

        @task()
        def extract_files():
            return {"source": "files", "rows": 80}

        extract_api()
        extract_db()
        extract_files()

    with TaskGroup(group_id="load_targets") as load_group:
        @task()
        def load_warehouse():
            print("Loaded into warehouse")

        @task()
        def load_cache():
            print("Loaded into cache")

        load_warehouse()
        load_cache()

    @task()
    def finish():
        print("Pipeline finished")

    # Dependencies wire to the TaskGroup itself, not to individual
    # tasks inside it - every task in extract_group finishes before
    # load_group's tasks can start.
    start() >> extract_group >> load_group >> finish()


multi_source_pipeline()

That's the exact DAG behind this real, successful run — both groups expanded, showing every task inside them:

Figure — TaskGroups render as bordered boxes in the Graph View. Click the box to collapse it back down to a single node.

Collapsed, extract_sources and load_targets shrink to two boxes total instead of five — that's the entire point. On a real 40-task DAG organized into 6-8 groups, the Graph View goes from unreadable to skimmable in about thirty seconds of refactoring.

Task IDs Inside a Group

Airflow automatically prefixes task IDs with the group ID, using a .:

This matters the moment you reference a task from outside its group — e.g., pulling an XCom: ti.xcom_pull(task_ids="extract_sources.extract_api"), not just "extract_api".

TaskGroups vs Dependency Behavior

It's worth being explicit about what TaskGroups don't change:

If you've seen older Airflow tutorials mention SubDAGs — TaskGroups are almost always the better, simpler replacement for what SubDAGs were trying to solve.

This content is reserved for Premium Members.